Apparatus and method for processing data stream

ABSTRACT

A camera controller (1) with an image data stream input (2) and an image data stream output (4), wherein a first image data stream path (3) and a second image data stream path (5) are formed. An image processing unit (6) is arranged in the second image data stream path (5). A switching unit (7) is controlled by a comparison unit (8). The comparison unit (8) compares a first checking data stream (9), which was diverted from an image data stream (1) upstream of the image processing unit (6), and a second checking data stream (10), which was diverted from the image data stream downstream of the image processing unit (6). The image data stream paths are switched via the switching unit (7) if the deviation of the second checking data stream (9) from the first checking data stream (10) exceeds a threshold value.

INCORPORATION BY REFERENCE

The following documents are incorporated herein by reference as if fullyset forth: German Patent Application No. 10 2019 109 189.6, filed Apr.8, 2019.

TECHNICAL FIELD

A method and apparatus for image processing, in particular in a cameracontroller, having an image data stream input, which is connectable tothe image output of an image recording apparatus, an image data streamoutput, which is connectable to an image display apparatus, a firstimage data stream path from the image data stream input to the imagedata stream output, and a second image data stream path from the imagedata stream input to the image data stream output, in which an imageprocessing unit is arranged.

BACKGROUND

It is known, in particular in endoscopy, to prepare image data streams.There is here a desire for increasingly more complex preparationalgorithms, for example for feature detection, to highlight specificimage contents by color, or to measure the features and superpose thedimensions in the image data stream, or for general image improvement,such as for enhancing contrast.

It is known to use a graphic processor unit, GPU, to this end so as tobe able to image these complex processing algorithms.

However, this gives rise to the problem that a sufficient failure safetymust be ensured. In particular in endoscopic applications during asurgery, tearing or failure of the image data stream must be avoidedunder all circumstances.

To this end, the prior art has proposed to monitor the GPU with the aidof a monitoring module and, if the proper function of the GPU can nolonger be ascertained, to bridge the image data stream path of the GPU.

To ascertain the correct function of the GPU, an additional monitoringunit that monitors critical parameters such as temperature and/or clockfrequency can be present.

For monitoring, it is also possible for example to transmit anadditional task to the GPU that the GPU must additionally work throughso that, if the result of this additional task is not produced, it ispossible to ascertain that the GPU no longer operates properly.

One problem is here that the GPU must reduce its output if it overheatsand thus less computing capacity for the actual image processing isavailable.

SUMMARY

It is now an object of the invention to provide a method and anapparatus that provide a simple, fast and reliable detection of thecorrect function of an image processing unit.

This object is achieved by a method with one or more features of theinvention.

The method according to the invention accordingly makes provision for afirst checking data stream to be produced, for a second checking datastream to be diverted from the image data stream in the second imagedata stream path, for a deviation between the second checking datastream and the first checking data stream to be ascertained, and for theimage data stream path to be switched over in dependence on thedeviation.

Producing a checking data stream can take place outside the imageprocessing unit (GPU), which is why no computing power of the GPU forprocessing additional tasks is necessary. A power reduction due tooverheating accordingly does not occur, ensuring uninterrupted imageprocessing. On the other hand, a malfunction can also be recognized veryquickly and independently of the image processing unit, with the resultthat if the GPU fails or in the case of an error of the GPU, fast andreliable switching-off or bypassing can take place.

The checking data stream can practically be any piece of informationthat is associated or may be associated with the image data, for examplea time signal or a timecode.

In an advantageous embodiment of the invention, the first checking datastream is diverted from an image data stream upstream of the imageprocessing unit.

The idea is here to divert a first checking data stream practicallydirectly from the input image data stream. The second checking datastream is diverted downstream of the image processing unit. If the imageprocessing unit functions without error, both checking data streamsshould be identical or at least similar. In the reverse, a greaterdeviation between the two checking data streams signifies a malfunctionof the image processing unit.

In an advantageous embodiment, a switch to the first image data streampath occurs if the deviation exceeds a threshold value. This thresholdvalue can be fixedly specified or be variably dependent on otherparameters. The first image data stream path bypasses the imageprocessing unit, which is why in the case of a failure or malfunction ofthis unit at least one raw signal or a rudimentarily processed imagedata stream is present at the image data stream output. Consequently, animage data stream for representation on an image display apparatus isavailable almost without interruption.

In an advantageous embodiment, the second checking data stream isascertained in the same way as the first checking data stream. In thisway, the two checking data streams can be simply compared to one anotherand differences can be ascertained.

The two checking data streams can be diverted for example directly fromthe respective image data stream path and be compared.

Due to the possibly different signal propagation times in the secondimage data stream path, however, it may be advantageous if the firstchecking data stream is stored in a buffer. The second checking datastream is here compared to the stored first checking data stream.

In an advantageous embodiment, the buffer is embodied in the form of aFIFO memory. The latter permits in a simple manner a comparison of thetemporal order, for example, of timestamps.

In a particularly advantageous embodiment of the invention, the firstchecking data stream is stored outside the visible image information inthe image data stream. In this way, a simple temporal assignment of thechecking data can take place.

The checking data stream can in particular be saved in a blankinginterval, preferably in the vertical blanking interval. It can also besaved in the pixel data of a covered region and/or inactive region ofthe image sensor. In this way, a negative impact on the image datastream is ruled out.

In an advantageous embodiment of the invention, the checking data fromthe first checking data stream stored in the image data stream areextracted from the image data stream downstream of the image processingunit as a second checking data stream and compared to the checking data,which are stored in the buffer, of the first checking data stream. Inthis way, it is particularly simple to ascertain a deviation between thetwo checking data streams.

In an expedient embodiment, the first checking data stream is a sequenceof timestamps that are assigned to the individual images of the imagedata stream. The deviation is here ascertained from the order andtemporal deviation of the timestamps in the second checking data stream.The timestamps are to this end saved not only in the buffer, but also inthe image data stream. If the image processing unit is defective, thetimestamp in the image signal is changed, falsified or removed. Thatmeans the second checking data stream is changed. A comparison of thesecond checking data stream to the first checking data stream in thebuffer thus permits fast and unique detection of a fault or a defect ofthe image processing unit.

In one development of the invention, in the image processing unit, thetimestamp is extracted at the beginning of each partial processing stepand added again to the processed image at the end of the partialprocessing step. In this way, it is not only possible to detect a defector an error of the image processing unit, but it is also possible todetect individual faulty processing steps. To this end, it may beadvantageous if the second checking data stream is compared to the firstchecking data stream after every processing step. In this way, it ispossible even before the image processing is complete to detect a defector an error and for a switch to the first image data stream path to betriggered. The switch accordingly takes place even faster, such that anyimage failure is shorter or does not occur at all.

In an alternative embodiment of the invention, the first checking datastream is a temporal profile of a brightness, of a histogram or of otherimage information of the image data stream. It has been shown that, eventhough such checking data streams change due to the image processing,basic properties, such as for example the temporal position of maximaand minima are maintained. To compare such checking data streams, it istherefore possible to calculate for example a correlation of the twochecking data streams.

Generally, a camera controller has an image control unit or arudimentary image processing unit, which prepares the raw data of animage sensor such that they can be guided to an image display apparatus.This image control unit or generally the image processing units of thefirst image data stream path generally have a design that is as simpleand cost-effective as possible.

In an advantageous embodiment of the invention, the second image datastream path is diverted from the first image data stream path upstreamof such an image control unit of the camera controller. This preventsnegative image influences on account of these simple image signalprocessing units from also being contained in the second image datastream and from no longer being removable there.

The invention also comprises a camera controller, having an image datastream input, which is connectable to the image output of an imagerecording apparatus, an image data stream output, which is connectableto an image display apparatus, a first image data stream path from theimage data stream input to the image data stream output, a second imagedata stream path from the image data stream input to the image datastream output, in which an image processing unit is arranged, aswitching unit with which an image data stream is switchable between thefirst image data stream path and the second image data stream path. Thiscamera controller is characterized according to the invention in thatthe camera controller has a comparison unit programmed as part of thecontroller, which is connected to the switching unit and which cancompare a first checking data stream and a second checking data stream,which was diverted from the image data stream downstream of the imageprocessing unit, and which can transmit a switching signal to theswitching unit such that a switch to the first image data stream pathoccurs if the deviation of the second checking data stream from thefirst checking data stream exceeds a threshold value.

In particular, the first checking data stream can have been divertedfrom the image data stream upstream of the image processing unit.

The camera controller preferably has a buffer, in particular a FIFOmemory, for storing the first checking data stream. Such a buffersimplifies the comparison of the checking data streams even in the caseof different propagation times of the two image data stream paths.

In particular, the camera controller is embodied and/or set up forperforming the method according to the invention.

The invention furthermore comprises a camera system, in particular avideo endoscope, with a camera controller according to the invention,with at least one image recording apparatus, with an objective lens, andwith an image sensor, which is connected to the image data stream inputof the camera controller, and with at least one image display apparatus,which is connected to the image data stream output of the cameracontroller.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be explained in more detail below on the basis of afew advantageous exemplary embodiments with reference to the attacheddrawings, in which:

FIG. 1: shows a flow chart of a performance of the method according tothe invention,

FIG. 2: shows a block diagram of a camera controller according to theinvention,

FIG. 3: shows a block diagram of a camera system according to a firstembodiment of the invention,

FIG. 4: shows a block diagram of a camera system according to a secondembodiment of the invention,

FIG. 5: shows a block diagram of a camera system according to a thirdembodiment of the invention,

FIG. 6: shows a schematic illustration of an image sensor, and

FIG. 7: shows a diagram of the brightness profiles of the image datastream before and after the image processing.

DETAILED DESCRIPTION

FIG. 2 shows a block diagram of a camera controller according to theinvention, which is denoted overall with 1.

The camera controller 1 has an image data stream input 2, which can beconnected for example to a video camera or to a camera head of anendoscope. The image data stream input 2 can receive for example animage data stream of an image sensor.

The image data stream input 2 is connected, via a first image datastream path 3, to an image data stream output 4. The image data streamoutput 4 is connectable for example to an image display apparatus, suchas a display screen.

The camera controller 1 has a second image data stream path 5, whichconnects the image data stream input 2 to the image data stream output4. However, an image processing unit 6 is arranged in the second imagedata stream path 5. This image processing unit 6 can have, for example,a graphic processor or a special image signal processor or a combinationof different processors. Using the image processing unit 6, compleximage processing tasks are able to be performed, for example featuredetection, image improvement and superposition of further information.

In the standard setting, the second image data stream path 5 and thusthe image processing unit 6 are active. According to the invention, aswitch to the first image data stream path 3 now takes place if theimage processing unit 6 is defective or if no image data stream ispresent, or only a defective image data stream is present, at the imagedata stream output 4 for a different reason.

To this end, in the example, the image data stream output 4 has aswitching unit 7, which is controlled by a comparison unit 8. Thecomparison unit 8 receives a first checking data stream 9, which isdiverted in the example in the image data stream input 3, and a secondchecking data stream 10, which is diverted in the image processing unit6. In the comparison unit 8, the two checking data streams are comparedand the deviation is ascertained.

The switch to the first image data stream path 3 is effected inaccordance with a method according to the invention if the deviationexceeds a threshold.

FIG. 1 shows a flow chart of an advantageous embodiment of a methodaccording to the invention.

In a first step 11, a first checking data stream 9 is diverted from theimage data stream. This can take place in the first image data streampath 3 or in the second image data stream path 5. At any rate, thediverting takes place upstream of the image processing unit 6. The firstchecking data stream 9 can for example be a sequence of timestamps or atemporal brightness profile or another piece of information associatedwith the image data stream.

In a second step 12, a second checking data stream 10 is diverteddownstream of the image processing unit 6 in the second image datastream path 5. The second checking data stream 10 is preferably divertedin the same way as the first checking data stream 9.

In the subsequent step 13, a deviation between the second checking datastream 10 and the first checking data stream 9 is ascertained. Dependingon the type of the checking data streams, the deviation can beascertained for example as the temporal deviation of timestamps, anincorrect order of the timestamps, or a deviating brightnessdistribution. It is also possible for example to ascertain a correlationof the checking data streams if a simple comparison is not possible.

In the subsequent step 14, the ascertained deviation is compared to athreshold value. If the ascertained deviation is greater than thethreshold value, then, in a switching step 15, for example using aswitching unit 7, a switch to the first image data stream 3 is performedand thus a defective image processing unit 6 is bridged or bypassed. Ifthe deviation is below the threshold value, the second image data streampath 5 is switched on 16 or maintained. The method is then repeatedcyclically.

The method is a continuous method, which begins with the switching-on ofthe camera controller 1 and ends only upon switch-off. The method canmoreover be synchronized with the image data stream input 2, such that,for example, a new cycle begins in each case after a complete image isreceived.

FIG. 3 shows a camera system 17 in accordance with a first embodiment ofthe invention. The camera system has a camera controller 1, an imagerecording apparatus 18, and an image display apparatus 19.

The image recording apparatus 18 has an objective lens 20 and an imagesensor 21, which is connected to the image data stream input 2 of thecamera controller. The image display apparatus 19 is connected to theimage data stream output 4 of the camera controller 1.

The camera controller 1 in this embodiment has an image processing unit39 in the first image data stream path 3. The image processing unit 39has a decoder unit 22 for preparing the image data of the image sensor21. Downstream of the decoder unit 22, a camera control unit 23 isconnected, which is connected, via a camera control line 24, to theimage recording apparatus 18 so as to control camera parameters such asexposure or the f-number. The camera control unit 23 also performsimage-sensor-specific adaptations, such as gamma correction orde-Bayering. Downstream of the camera control unit 23, a switching unit7 is connected, which here implicitly also has a comparison unit 8 inaccordance with FIG. 2, which receives and compares checking datastreams.

Due to the switching unit 7, a switch to a second image data stream path5 can be performed, in which an image processing unit 6 for compleximage processing is arranged. The second image data stream path 5 isalso led back in the switching unit. The downstream screen adaptationunit 24 regulates for example brightness, contrast and image sharpnessof the image. Finally, the image processing unit 39 has an encoder unit25, which adapts the image signal to conventional interfaces such asHDMI, DisplayPort or SDI. The encoder unit 25 is connected to the imagedata stream output 4. The image processing unit 39 is preferablyembodied as a logic module, in particular as an FPGA, wherein all unitscontained are contained in the logic module.

FIG. 4 shows a camera system 17 having a camera controller 1, whichsubstantially corresponds to the camera controller of FIG. 3. However,in this embodiment, the second image data stream path 5 already divertsafter the decoder unit 22. The components arranged in the first imagedata stream path 3, in particular the camera control unit 23, containsonly rudimentary image processing functions for cost reasons. For thisreason, losses in terms of image quality and other image artifacts mayoccur here. Due to the previous diverting of the second image datastream path 5, it is avoided that these image errors are also containedin the image data stream, which was prepared with much outlay, of theimage processing unit 6 and can no longer be removed therefrom. Theswitching unit 7 in this embodiment is not arranged until after thescreen adaptation unit 24.

FIG. 5 shows a camera system 17 having a further embodiment of a cameracontroller 1. This camera controller 1 corresponds in the first imagedata stream path to the camera controller 1 of FIG. 3. The second imagedata stream path 5 is diverted downstream of the decoder unit 22 in thisembodiment, too.

The decoder unit 22 supplies a VSync signal 26 to a timestamp generator27. The timestamps are stored as a first checking data stream 9 in aFIFO memory module 28. Even in this embodiment, the image processingunit 39 is preferably embodied in the form of an FPGA, wherein inparticular also the FIFO memory module 28 can be formed within the FPGA.

The embodiment shown additionally has a first video analysis unit 29, inwhich characteristic properties of the image data stream areascertained, for example a brightness profile 38 (see FIG. 7). In ametadata unit 30, the image data stream is extended by a metadataregion. This is typically done in a blanking interval of the image datastream or in the pixel data of an unused region of the image sensor. Ina checking data unit 31, the video analysis is written as a furtherfirst checking data stream 9 of the video analysis unit 29 together withthe appropriate timecode into the metadata region of the image datastream before the image data stream is passed to the image processingunit 6.

Downstream of the image processing unit 6, in a first extraction unit32, the timestamp is extracted from the metadata region of the imagedata stream as a second checking data stream 10. In a comparison unit 8,which is connected to an output of the FIFO memory 28, said secondchecking data stream 10 is compared to the first checking data stream 9from the FIFO memory 28 and a deviation is ascertained. The comparisonunit 8 is connected to the switching unit 7 so as to perform a switchthere if the deviation exceeds a threshold value.

In addition, the camera controller 1 shown has a second extraction unit33, which extracts the video analysis data as a further first checkingdata stream from the metadata of the image data stream. A second videoanalysis unit 29′ performs the same video analysis on the processedimage data stream in parallel, which produces a further second checkingdata stream. In an analysis comparison unit 34, the two video analyses,i.e. checking data streams, are compared. The analysis comparison unit34 is likewise connected to the switching unit 7, with the result that aswitch can also be performed if the video analysis data deviate.

FIG. 6 shows by way of example an image sensor 21. The image sensor 21has an active region 35, which is smaller than the area of the imagesensor 21. Around the active region, there is a small passive region 36,which is illuminated but not used. Around said passive region 36, acovered region 37, which contains covered pixels, is present. Thisregion is due to the time raster, which is the result of the pixelclock. The pixel clock produces more pixels per frame than the sensorhas image points. The result of this is that not all the time rasterrows and not all the image raster columns have active image points. Themetadata of the image data stream can be saved for example in the pixeldata of the covered region 37 and/or of the passive region 36, as aresult of which the data capacity is significantly greater with respectto the blanking interval.

FIG. 7 shows by way of example a temporal brightness profile 38 of animage data stream, as is established for example by the first videoanalysis unit 29. The dashed line represents the temporal brightnessprofile 38′ of the processed image data stream, as is established forexample by the second video analysis unit 29′. Due to the imageprocessing unit 6, the amplitude of the brightness profile 38 haschanged, but the minima and maxima are maintained temporally. In thecase of a defective image processing unit 6, for example the temporalpositions of the minima and maxima would change. A deviation that can beused for the switch in the switching device 7 can be ascertained fromthe phase position of the two brightness distributions.

LIST OF REFERENCE SIGNS

-   -   1 Camera system    -   2 Image data stream input    -   3 First image data stream path    -   4 Image data stream output    -   5 Second image data stream path    -   6 Image processing unit    -   7 Switching unit    -   8 Comparison unit    -   9 First checking data stream    -   10 Second checking data stream    -   11 Method step 1    -   12 Method step 2    -   13 Method step 3    -   14 Method step 4    -   15 Method step 5    -   16 Method step 6    -   17 Camera system    -   18 Image recording apparatus    -   19 Image display apparatus    -   20 Objective lens    -   21 Image sensor    -   22 Decoder unit    -   23 Camera control unit    -   24 Screen adaption unit    -   25 Encoder unit    -   26 VSync signal    -   27 Timecode generator    -   28 FIFO memory    -   29 First video analysis unit    -   29′ Second video analysis unit    -   30 Metadata unit    -   31 Checking data unit    -   32 First extraction unit    -   33 Second extraction unit    -   34 Analysis comparison unit    -   35 Active region    -   36 Passive region    -   37 Covered region    -   38 Brightness profile    -   38′ Brightness profile of the processed image data stream    -   39 Image processing unit

1. A method for image processing, the method comprising: providing acamera controller (1) having an image data stream input (2) which isconnectable to an image recording apparatus (18), an image data streamoutput (4) which is connectable to an image display apparatus (19), afirst image data stream path (3) from the image data stream input (2) tothe image data stream output (4), and a second image data stream path(5) from the image data stream input (2) to the image data stream output(4), in which an image processing unit (6) is arranged; producing afirst checking data stream (9); diverting a second checking data stream(10) from the image data stream in the second image data stream path(5); ascertaining a deviation between the second checking data stream(10) and the first checking data stream; and switching over the imagedata stream path in dependence on the deviation.
 2. The method asclaimed in claim 1, wherein the switching over to the first image datastream path (3) occurs if the deviation exceeds a threshold value (14).3. The method as claimed in claim 1, wherein the first checking datastream (9) is diverted (11) from an image data stream upstream of theimage processing unit (6).
 4. The method as claimed in claim 1, whereinthe second checking data stream (10) is ascertained in a same way as thefirst checking data stream (9).
 5. The method as claimed in claim 1,further comprising storing the first checking data stream (9) in abuffer.
 6. The method as claimed in claim 5, wherein the buffer is aFIFO memory (28).
 7. The method as claimed in claim 5, wherein the firstchecking data stream (9) is stored outside from visible imageinformation in the image data stream.
 8. The method as claimed in claim5, wherein the first checking data stream (9) is stored in at least oneof a blanking interval, a vertical blanking interval, or a passiveregion (36) or a covered region (37) of an image sensor (21).
 9. Themethod as claimed in claim 8, further comprising extracting the checkingdata from the first checking data stream (9) stored in the image datastream from the image data stream downstream of the image processingunit (6) as a second checking data stream (10) and comparing the secondchecking data stream (10) to the checking data, which are stored in thebuffer, of the first checking data stream (9).
 10. The method as claimedin claim 9, wherein the first checking data stream (9) comprises asequence of timestamps that are assigned to the individual images of theimage data stream and the deviation is ascertained from an order andtemporal deviation of the timestamps in the second checking data stream.11. The method as claimed in claim 10, wherein in the image processingunit the timestamp is extracted at a beginning of each partialprocessing step and added again to a processed image at an end of thepartial processing step.
 12. The method as claimed in claim 1, whereinthe first checking data stream (9) is a temporal profile of a brightness(38), of a histogram or of other image information of the image datastream.
 13. The method as claimed in claim 1, wherein the second imagedata stream path (5) is diverted from the first image data stream path(9) upstream of a camera control unit (23) of the camera controller (1).14. A camera controller (1), comprising: an image processing unit (6),an image data stream input (2) which is connectable to the image outputof an image recording apparatus (18), an image data stream output (4)which is connectable to an image display apparatus (19), a first imagedata stream path (3) from the image data stream input (2) to the imagedata stream output (4), a second image data stream path (5) from theimage data stream input (2) to the image data stream output (4), inwhich the image processing unit (6) is arranged, a switching unit (7)with which an image data stream is switchable between the first imagedata stream path (3) and the second image data stream path (5), acomparison unit (8) connected to the switching unit (7) configured tocompare a first checking data stream (9) and a second checking datastream (10), which was diverted from the image data stream downstream ofthe image processing unit (6), and which can transmit a switching signalto the switching unit (7) such that a switch to the first image datastream path (3) occurs if a deviation of the second checking data stream(9) from the first checking data stream (10) exceeds a threshold value.15. The camera controller (1) as claimed in claim 14, wherein the cameracontroller (1) has a buffer that is configured to store the firstchecking data stream (9).
 16. A camera system (17) comprising the cameracontroller (1) as claimed in claim 14, at least one image recordingapparatus (18), with an objective lens (20) and an image sensor (21),connected to the image data stream input (2) of the camera controller(2), and at least one image display apparatus (19) connected to theimage data stream output (4) of the camera controller (1).